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assignments that require students to develop research questions; (3) articles 
reviews; and (4) annotating output from computer runs. Examples of each 
assessment type are given. At the end of the course, the 14 students were 
asked to rate each strategy. Overall, the survey findings suggest that a 
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Approaches to Assessment in Multivariate Analysis 



1. Introduction 

Recently, a student came to my office to discuss her research study in the area of sports- 
fitness. A master’s level student, she had collected data on several leg and thigh variables and 
wanted to assess the relationship between these measures and later occurrence of injury among 
women athletes. An interesting study, certainly appropriate for a master’s level thesis project. 
Her question? How to interpret the output. The real problem? The student wasn’t sure what 
statistical test to use, so she decided to “click on all the buttons” to make sure she got the right 
test! 



I know this is not an unusual situation for those of us who teach educational statistics 
courses. In fact, ten years ago Searle (1989) and Dallal (1990) discussed just this kind of 
situation in their debate on the merits of including instruction on statistical computing packages 
as part of statistics courses. However, while there was clearly a missing link for this student 
hindering her ability to move from the research question to an appropriate analysis and 
interpretation of the data, I feel that this scenario speaks to a larger gap between how we assess 
learning in statistics courses and the actual statistical competencies - knowledge and skills - that 
we expect our students to have after completing these courses. 

For graduate students in the social sciences, coursework in advanced statistical methods 
should enable them to embark on a successful academic career. To do this, students need 
sufficient preparation in statistical methods for completion of a culminating project (thesis or 
dissertation) which establishes a personal contribution to their field of study, and for making 
continued contributions to their field through an active and productive research agenda. We 
need to identify effective activities and assessments that can promote learning for transfer to 
research experiences outside of our classrooms. 

2. Purpose 

The purpose of my presentation today is to review trends in assessment in quantitative 
courses and illustrate several options and approaches to assessment for advanced courses at the 
graduate level, particularly in multivariate analysis. Hopefully, this presentation will stimulate 
interest and further discussion among statistics educators, and encourage others to pursue 
empirical studies in this area. I present the results of a survey of student ratings of assessment 
strategies I’ve used in my own courses, including 

(a) structured data analysis assignments 

(b) open-ended assignments 

(c) article reviews, and 

(d) annotating computer output for specific multivariate procedures. 

3. Trends 

In recent years, one of the most significant trends to emerge in statistical education 
involves an emphasis on activity-based learning. Activity-based learning can take many forms, 
including cooperative group tasks, computer based problem-solving activities or simulations, or 
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the collection and analysis of “real,” as opposed to textbook, data. One goal of these reforms is 
to improve statistical literacy, which is the ability to understand and discuss statistics 
normatively. Learning can be enhanced when students are actively engaged in a problem, and 
are able to structure their new knowledge in terms of their own experiences and academic 
interests. In my own courses, I’ve tried to include an assortment of assessment activities with a 
view towards supporting the development of statistical literacy. 

Actually, I think “statistical clarity” might be a better term for what I want students to 
achieve in my classes, since it more strongly emphasizes several inter-related goals in 
multivariate analysis - clarity of purpose in research design and analysis, and clarity in 
articulation of results. By assessing the knowledge, skills, and abilities we feel to be important, 
students will be in a better position to appreciate and review their own efforts at learning. In 
fact, the National Council of Teachers of Mathematics (NCTM) has recently published 
Assessment Standards (1995) for school mathematics which are every bit as applicable to the 
graduate classroom as well. Their recommendations include viewing assessment as a 
convergence of information from different kinds of sources or activities; that assessment should 
motivate greater achievement; and that connections among different kinds of knowledge should 
be assessed. 

4. Assessment as Learning 

From the NCTM recommendations, it is clear that statistics educators need to move 
towards creative and authenticated approaches to assessment of student learning in statistics. 
Garfield (1994; Gal & Garfield, 1997) has often reminded us to use assessment as a vital part of 
the learning process and not just as a grading strategy. The tasks that we give our students today 
should result in learning that can transfer to new situations and research settings tomorrow. This 
goal also is emphasized in the Assessment Standards (1995) since, “assessment should be a 
means of fostering growth toward high expectations” (p. 1). In the multivariate classroom, the 
expectation typically is for students to consolidate conceptual and procedural knowledge about 
advanced statistical techniques. How this can best be accomplished remains an issue for 
consideration by educational researchers and instructors of quantitative courses. 

5. Background: Training in Graduate Statistics 

The literature on pedagogy in introductory statistics courses continues to grow (e.g. ISR 
special issue, April 1995; JEBS special issue, Spring 1996; Gal & Garfield, 1997), yet 
comparatively little information is available on how well or how much we train our graduate 
students beyond the introductory level. There is evidence that graduate training in statistics 
needs to be improved, particularly for substantive researchers whose areas of expertise may not 
be quantitative methods Keselman, Huberty, Lix, et al., 1998). For example, Keselman and his 
colleagues found that published research in respected journals in education and psychology often 
contains no reference to issues such as verification of assumptions, power, or effect size (1998). 

Despite the fact that relatively little published work is available on assessment in 
advanced statistics courses such as multivariate analysis, we can build on the suggestions of 
statistics educators and researchers who have concentrated on introductory statistics courses in 




order to inform better practice in instruction and improve learning in advanced courses. In 
particular, statistics educators argue for a more active involvement on the part of the student, 
strengthening their investment in their own learning processes. 

6. Background: Transfer 

There is still a lot we don’t know about transfer of knowledge to new situations (Pintrich, 
1994), yet transfer is clearly an important goal for courses in multivariate analysis. Important 
outcomes for an advanced statistics class are for students to understand published research 
literature in their field and be able to apply the principles and techniques they’ve learned to new 
and perhaps rapidly changing research initiatives and directions. 

Familiar early studies by Tversky and Kahnemann (1983) and Shaughnessy (1981) 
showed that formal instruction in statistics did not protect people from their pre -course 
misconceptions in statistical reasoning or improve their ability to make informed decisions under 
uncertainty. Results of other studies by Fong, Krantz, and Nisbett (1986) and their colleagues 
(19xx) have offered some limited support for transfer of statistical training. More recently, 
research evidence supports the idea that conceptual knowledge plays an important role in the 
acquisition, development, or use of procedural knowledge (O’Connell, 1999; Rittle-Johnson & 
Alibali, 1999), but how the relationship between these two kinds of knowledge affects transfer to 
new situations needs further attention. Finally, in terms of transfer of statistical knowledge and 
skill to actual research activities, Keselman et al. (1998) report that “a substantial gap exists 
between the inferential methods that are recommended in the statistical research literature and 
those techniques actually adopted by applied researchers” (p. 351). 

7. Background: Valuing what we Assess 

One way we can begin to strengthen the bond between conceptual knowledge and 
procedural knowledge in advanced statistics is to review our own instructional practices, 
including the goals for intended student learning and our corresponding assessment strategies. In 
particular, assessment that is grounded as much as possible in real-world experiences [offers the 
best possibility for] can guide our teaching for transfer of statistical knowledge to novel 
situations and research questions. One of Garfield’s principles on statistical learning is that 
students will come to value what they know will be assessed (1995). If we can develop our 
assessments so that students learn to appreciate how statistical methodologies may be applied to 
diverse data sets or research situations, then we clearly will be moving towards strengthening our 
graduate level training in advanced statistical analysis courses. 

8. Multivariate Assessment Study 

My findings provide a summary of how one researcher (the author) has used alternatives 
to traditional methods of assessment in a course on multivariate analysis. As the literature is 
sparse in this area, one way to begin to understand the impact that assessment has on student 
learning of multivariate techniques is by studying student reactions to particular forms of 
assessment. This approach is helpful for reflecting on what our own goals might be for our 
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students, understanding how well these goals are being met, and suggesting methods for 
improving both teaching and learning of advanced statistics. 

This is the first course in a two semester sequence in multivariate analysis. Course topics 
include: 

• Multiple regression (review and additional material) 

• Multivariate analysis of variance and covariance 

• Factorial analysis of variance 

• General linear model 

• Repeated measures analysis and profile analysis 

• Discriminant function analysis 

• Canonical Correlation 
. PC A/FA 

The course objectives are: 

(1) develop the ability to work competently with a variety of multivariate statistical 
techniques; 

(2) understand research literature/reports involving these techniques; 

(3) use the computer (SPSS) to run and interpret the results of multivariate analyses; and, 

(4) write up coherent results from these analyses. 

9. Assessments Used 

I used four different kinds of assessments during my course: structured data analysis 
assignments where specific questions are posed and answers found through computer analysis or 
follow-up hand computation when necessary; open-ended assignments which requires students to 
develop a research question based on a particular data set, analyzing the data appropriately for 
the question and the data, and interpreting/writing up a formal report of the results; article 
analysis which requires students to read a particular research article and answer questions based 
on the article; and annotating output from computer runs. Examples for each assessment type are 
provided below. 

Structured data analysis assignments are those where I supply the data, and sometimes the 
program syntax, and ask specific questions for the students to answer. Generally, all students in 
the class should have the same or very similar responses to each question. 

Example: Multiple Regression Review 

The data consists of four variables collected on a small sample of junior high school students 
(adapted from Tatsuoka, 1988). The dependent variable of interest is an achievement test score 
in the physical sciences. A teacher is interested in studying the extent to which achievement in 
the physical sciences (Y) can be predicted from measures of math ability (XI), mechanical 
reasoning (X2), and/or creativity (X3). To begin to address the teachers concerns, we are going 
to run a series of three regression analyses, keeping the following questions in mind: 




6 



1. After controlling for differences in math skill within this sample of students, does mechanical 
reasoning significantly improve our predictions of achievement? How much more explanation 
of variability in the physical science measure do we achieve by including mechanical reasoning 
in a model that already contains math skill? 

2. Over and above the explanation obtained from the first two variables, should we include the 
creativity test score in our prediction equation? Discuss the evidence as to why or why not. 
Based on all of your analyses, what suggestions/conclusions would you give the teacher in terms 
of variables affecting achievement in the physical sciences? What model would you recommend 
the teacher use for predicting achievement in the physical sciences? What concerns would you 
share with the teacher about the data and the results? 

Open-ended Assignments. For these assignments, I have several data sets available for students 
to consider establishing an appropriate research question for and reporting on their results, 
including High School and Beyond; automobile fatality rate data (Judd & McClelland, 1989); a 
survey on attitudes/perceptions regarding distance education (King, 1999); behavioral variables 
related to condom use among women (Stark, Tesselaar, O’Connell, et al., 1996); relationships 
among different kinds of coded errors during probability problem-solving (O’Connell, 1999); 
and data from the evaluation of an adolescent pregnancy prevention project (O’Connell, 1997). 

Example: Multivariate Project 

The syllabus gives a short summary of goals for the project. I also provide an outline of what the 
project paper should look like. Briefly, the goals are to establish a research question appropriate 
to one of the multivariate techniques we are learning (and corresponding to the chosen data set), 
and prepare a report detailing the methods used to answer your research question and 
interpretation of results. 

• Short, independent, research and analysis paper, limited to 5 to 10 pages. 

• Written in APA style (or style appropriate for your field). 

• Purpose of the project is to show me that you can understand and write intelligently about 
a specific multivariate technique (one of the above). 

• Several data sets are available for analysis. 

• If you want to use a data set from a project you are working on, you must clear it with me 
first. The data must have already been collected, you need permission from the PI, and 
any Human Subjects issues must already be addressed. I reserve the right to say yes or 
no to people’s request to use their own data, since I want people to be successful at their 
particular analysis and not bogged down by potentially messy or complicated data sets. 

• Project Data Information Sheet is to be completed (a downloadable form) - describes 
variables and how they are measured. 

• Be aware that the analysis you might choose for the purposes of this course may not be 
the only approach for a particular data set/research question. I am interested in how you 
approach the particular analysis you selected. 

• You will be graded on the clarity of your analysis, and its correspondence to the research 
question you asked for the purpose of this course. 
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• The appendix to your project/paper should contain the output you used for analysis, 
annotated to describe how you used the output to address your research question(s). 

Article analysis. For the article review assignments, I ask the students to prepare a paper 
detailing the responses to several questions based on the research article. 

Article 1. Dant, R.P., Lumpkin, J.R., and Bush, R.P. (1990). Private physicians or walk-in 
clinics: Do the patients differ ? Journal of Health Care Marketing. 10 (2). 25-35. (also from 
Hair, et al„ (1992), p. 138-152). 

Article 2. Karacostas, D. D. & Fisher, G.L. (1993). Chemical dependencies in students with and 
without learning disabilities. Journal of Learning Disabilities. 26 (7) 491-495. 

Example: Karacostas & Fisher article 



1. Purpose 

2. Population/Sample - generalizing/validity issues? 

3. Variables - how measured? 

4. What are the research questions? What analyses were used to investigate these research 
questions? 

5. Assumptions? 

6. In Table 1, second entry is 34/38. Describe what both these values mean. 

7. Explain the canonical correlation and how it relates to % variance. 

8. Do you agree with their results/conclusions? Any limitations, even other than they mention? 
Missing information? 

9. Recommend any further analyses which might be appropriate here. What would be the 
hypotheses for these analyses if the study was replicated? 

Annotated Output. For these assignments, I ask students to run a particular analysis (after we 
had already reviewed the topic and an analysis in class), such as DFA, and to label on the output 
the elements necessary for understanding the results. Often, the programs may correspond to 
data available in the text (we used Stevens (1996) during the Fall 1999 semester). I also required 
them to annotate the output for their open-ended projects. 

Example: Factorial Manova, data suggested from Stevens page 310, #1. 

I changed some of the data points and made the data available on the webboard. 

a. run the analysis. 

b. annotate the output. 

c. state the assumptions and how you would verify them. Also write out the null and 
alternative hypotheses for the HOC test and report the results of this test. 

d. write a brief summary (a few paragraphs). Be sure to explain how your support or 
non-support of the assumptions required for Manova affect your interpretation of the 
results. 

e. give the effect size for any significant multivariate or univariate results that you report. 



1 0. Student Assessment Ratings 

At the end of the course, students were asked to rate each different assessment strategy in 
terms of (a) difficulty, (b) appropriateness to their needs while learning multivariate statistics, 
and (c) how well they felt they learned using that kind of assessment (n=14). I also asked 
students for additional comments regarding their likes and dislikes about each assessment 
strategy, suggestions for how to improve the use of that strategy, and any other observations or 
comments they might have. In terms of establishing a course grade, the two open ended 
assignments (midterm and final) were 25% each; the other assignments were averaged and worth 
50% total. 

In the tables that follow, I present the results for each assessment strategy, and highlight 
the model response for each technique. Other interesting findings are highlighted as well. The 
median rating for each assessment technique is presented in the last row of the tables. 

9. Difficulty Ratings 





Structured 

Computer 

Assignments 


Open-Ended 

Assignments 


Article 

Analysis 


Annotating 

Output 


(1) Not at all 
difficult 


7.1% 


0.0% 


7.1% 


0.0% 


(2) Slightly 
difficult but not 
too challenging 


35.7% 


14.3% 


28.6% 


57 . 1 % 


(3) Difficult but 
challenging 


50 % 


85 . 7 % 


50 % 


42.9% 


(4) Too difficult 


7.1% 


0.0% 


14.3% 


0.0% 


Median rating 


(3.0) 


(3.0) 


(3.0) 


(2.0) 



11. Appropriateness 





Structured 

Computer 

Assignments 


Open-Ended 

Assignments 


Article 

Analysis 


Annotating 

Output 


(1) Not at all 
appropriate or 
useful 


0.0% 


0.0% 


7.1% 


0.0% 


(2) Slightly 
appropriate but 
not for all my 
needs 


14.3% 


7.1% 


35 . 7 % 


35.7% 


(3) Appropriate 
for many of my 
needs 


78 . 6 % 


50 . 0 % 


35 . 7 % 


64 . 3 % 


(4) Very 
appropriate for 
all my needs 


7.1% 


42.9% 


21.4% 


0.0% 


Median rating 


(3.0) 


(3.0) 


(3.0) 


(3.0) 



12. Learning 





Structured 

Computer 

Assignments 


Open-Ended 

Assignments 


Article 

Analysis 


Annotating 

Output 


(1) Didn’t learn 
anything 


0.0% 


0.0% 


7.1% 


0.0% 


(2) Learned a 
little bit 


21.4% 


7.1% 


42 . 9 % 


28.6% 


(3) Learned 
enough to be 
comfortable 
with the topic 


57 . 1 % 


50 . 0 % 


28.6% 


57 . 1 % 


(4) Learned a 
great deal - 
more than I 
would have 
thought 


21.4% 


42 . 9 % 


14.3% 


14.3% 


Median rating 


(3.0) 


(3.0) 


(2.0) 


(3.0) 
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13. Preference Scores 

Students were also asked to indicate their order of preference among the four assessment 
strategies, with 1 = most preferred form of assessment, and 4 = least preferred. Average 
preference ratings indicated that the most preferred forms of assessment were the structured 
computer assignments (X=2.00), followed by annotating the output (X=2.31), use of open- 
ended assignments (X=2.38), and article analysis (X=3.15). Yet in terms of difficulty level, 
appropriateness, and learning ratings, the open-ended assignments actually received the better 
ratings overall. Approximately 86% of the students found the open-ended assignment 
challenging; 93% found the open-ended assignment appropriate for some or all of their needs; 
and 93% reported learning enough either to be comfortable with the topic or that they learned 
more than they would have thought. 

14. Why the switch between preferences and ratings? 

One explanation for the preference switch may be the time-involvement required for the 
open-ended assignment. Basing data analysis within the context of an actual research problem, 
as in the open-ended assignments, can be a strong motivator for students, and the experience 
requires the student to become an active participant in their own learning process. The structured 
assignments, with questions already posed by the instructor, provides greater direction and may 
have less cognitive load so in a sense might be considered easier. However, these structured 
assignments allow less flexibility and creativity in working towards a solution. Their purpose is 
very different from that of the open-ended assignments - although interpreting output is an 
important step towards interpretation and the convergence of procedural and conceptual 
knowledge. 

15. Student Comments 
Structured Computer Assignments: 

• “These smaller assignments helped me to better get into the topic, realize where I have 
difficulties and learn from your comments.” 

• “Doing this type of assignment first helped me with the less structured assignments 
later.” 

• “Written feedback was very useful.” 

Open Ended Assignments: 

• “This was difficult, but very necessary for us as competent researchers.” 

• “The open-ended nature of the assignments “forced” me to think about the bigger 
conceptual issues. It helped to apply the info to real-life examples, like I will be doing 
with my dissertation.” 
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• “Excellent, the openness addressed my own misconceptions.” 

Article Analysis: 

• “This assignment was time consuming b/c there was a lot of English vocabulary I have 
not been exposed to.” 

• “Would have liked to instead answer all of the questions and then discuss the article in 
class - but we didn’t have enough time.” 

• “The topic was not of personal interest.” 

Annotating Output 

• “This assignment was not clear to me so I did not like it or see its usefulness.” 

• “Excellent strategy; helped to really connect concepts to application.” 

• “We should be asked to annotate outputs more frequently.” 

16. What I’ve learned. 

Reflecting on student perceptions of the assessment strategies we use in our classes is a 
good first step towards improving our instruction and making our assessments more meaningful. 
Since our students will most likely be using the multivariate techniques they are learning as they 
begin their own academic careers, it is important not to loose site of their needs, and to ensure 
that accurate learning has occurred. In addition, the process of evaluating student perceptions of 
classroom activities models good strategies for potential future teachers of a complex topic such 
as multivariate analysis. 

In general, my findings suggest the following: 

• A variety of assessment strategies provides the opportunity for different kinds of 
feedback about the material being learned. 

• The assessment strategies we choose (and it is our choice, so we have to do it 
responsibly) can have a direct impact on student affect about the course. Since we 
would like students to get enjoyment from their effort - particularly for understanding 
their own data/research - this affect factor can’t be ignored. 

• Students are willing to be creative, try new forms of assessment, and also are eager to 
help us learn what works best. 

For my own course, this process has helped me think about how to refine and refocus my 
assignments for better student learning outcomes. 

• Need additional, and perhaps briefer structured assignments, to provide an opportunity 
for more feedback. 
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• Even with open-ended assignments, I can provide better guidance on what my 
expectations are. A project time-line, perhaps with a checklist for project milestones 
(research question, brief lit-review, analysis method, etc.) and a variable summary 
check list might enhance the learning process. 

• I think article critiques are important, and often form a part of comprehensive 
examinations. However, suitable class time needs to be reserved for review of these 
articles as a class. Perhaps consider group presentations of different articles. 

• Annotating output and writing up corresponding results sections can mimic the self- 
explanation strategies which are so important to good learning (Chi, et al., 1989, 
1994). 

• Still working on the best “mix” of assessments, but this personal study has provided 
me with a wealth of information on where I can improve my own instruction and 
assessment practices. 

o individual or group projects (Dietz, 1993; Cobb, 1992; Garfield, 1993; 

Holmes, 1997; O’Connell, 1997) 
o poster session displays (Denson, 1992) 
o oral assessment (Joliff, 1997) 
o creative multiple choice (Wild, 1997). 
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